DeLiberations Assessment Survey

Interim results - July 26th 1996

A survey was published on DeLiberations in May 1996, asking respondents to reply to nine questions aimed to provoke debate on assessment in universities and colleges. This report covers responses received by July, 1996 from nineteen recipients, twelve from the UK and one each from Sweden, Italy, Finland, USA, Canada, Germany and Australia.
[ Full list of respondents ]

The Survey Questions:

Assessment is the prime motivator for student effort: it's what makes them work hard
Exams are one of the least successful methods of assessing students ever devised
We don't have time any more to assess student essays: we need to find other more cost-effective means of assessment
Multiple-choice questions can only ever test very low-level educational objectives
The best person to assess student work must always be the tutor
It isn't possible to assess students fairly in groups, because the idle get carried by the rest and the best students get their marks pulled down by the weakest
The introduction of modular and semesterised university systems has irreparably damaged the student assessment process, with ten agreeing and nine disagreeing
Technology can help us assess students better
Other comments and issues for discussion

1. Assessment is the prime motivator for student effort: it's what makes them work hard

Fourteen agreed with the statement and five disagreed.

Sample comments of those who agreed with the statement were:

"Many students only seem prepared to make an effort if there is a tangible return: they'll do something for us if we'll do something for them. They have a very materialist view of University activity, education for its intrinsic interest seems very low on their priorities. A consequence is many individual items of assessment and elaborate overall assessment criteria. It adds to the considerable workload of staff in a way that should be unnecessary and narrows a student's view of the educational process"
"This is the crime that our education structure commits on our students today. Learners lose the sheer excitement of learning through the structures imposed on them by the formal curriculum and the school today."
"In my experience at post-graduate level, I have found students increasingly focused on what has to be done to pass exams. This is not just a tendency in education but also within business - just look at the performance related pay schemes and their impact on the earnings of executives"
"Doesn't necessarily motivate. The students have hoops to jump through"

Of those who disagreed, sample comments included:

"Assessment is a prime motivator for students to play a game with teachers"
"The ...statement considers extrinsic motivation only. This is not adequate from my point of view. Intrinsic factors are to be considered as well. Even if extended with the prefix self-(assessment) the statement is not to be accepted."

Survey Questions

2. Exams are one of the least successful methods of assessing students ever devised

Eleven agreed and eight disagreed with this statement.

Examples of what those who agreed with this statement said included:

"I think that we have better forms of examination now, especially open book exams. I feel that plagiarism is a real issue with course work and so I reluctantly go along with examinations"
"Exams suit some people and not others. The main problem with exams is not their use as an assessment tool, but their value or lack of it, to students, the majority of whom will cram for the exam and then proceed to forget what they have 'learnt' approximately 2.78 seconds after leaving the exam, just slightly quicker than a goldfish."
"Yes, but they are easy to administer"

Some who disagreed thought:

"Exams can assess a broad range of skills if well done. They are easy to do poorly. They have the advantage of being reasonably successful in ensuring that work for assessment is a student's own. don't forget that there are many exam styles, not just closed book"
"It depends on the design of the exam, but even multiple choice exams do have at least a methodological basis. But MC questions should always be in company with e.g. open questions. Furthermore is the fact that what is to be assessed an important question"
"In Italy, at university level, exams are the only form of evaluation but I don't think it is a successful method"

Survey Questions

3. We don't have time any more to assess student essays: we need to find other more cost-effective means of assessment

Nine concurred with the statement but ten disagreed.

Sample agreement statements were:

"If Unis weren't so badly underfunded then the staff student ratios would allow essays as a means of assessment"
"Essays also tend to be related to theoretical or hypothetical situation rather than be related to real practice"
"I have said that I agree with this. That reflects what actually happens and not what I do. I do try to use summative coursework in a formative sense and give fairly extensive feedback. Many colleagues don't. What is disturbing is that many students are more interested in the mark obtained than in reasons why, or ways of improving next time. We do need to find a better range of more-appropriate coursework assessments. Essays are not the be-all and end-all"

Sample disagreement comments included:

"An essay is a good way of assessing a student's skills in research, analysis and evaluation of a problem. It give a student time to make considered points - they don't often have such time in other types of assessment"
"We need to make the time, proper assessment of proper tasks is always time-consuming"
"Student essays are important from different points of view. Writing an essay always means creating a structure. It allows the discussion of complex problems that are more important in everyday life and in working life. Language, especially written language, is a real important factor concerning thinking and learning "

Survey Questions

4. Multiple-choice questions can only ever test very low-level educational objectives

Almost equal numbers were recorded for the statement with ten agreeing and nine disagreeing.

One who agreed said:

"They are also inappropriate generally at higher levels of education"

Comments in disagreement included:

"Unfortunately this is the entrenched concept associated with multiple-choice tests. It is decidedly not true. A well developed assessment vehicle can assess all levels of understanding and cognition associated with educational objectives and taxonomies"
"Well designed multiple-choice questions can be very effective in testing certain aspects of both knowledge and understanding. The Open University has used them to effect for many years. They are, however, very difficult to design, and are not necessarily an easy option for the assessor at least in the short term"
"I have seen some very sophisticated mqas, indeed a colleague here has developed many. But what worries me is that students don't see mqas this way; it does seem to drive them to adopt surface learning even when they know that understanding of concepts and principles and principles in action will be tested"

Survey Questions

5. The best person to assess student work must always be the tutor

Only six respondents agreed while thirteen disagreed.

One who agreed said:

"Like most of these questions, I want a slider, not radio buttons. Some competence-based skill activity can sensibly be assessed by technicians. Elementary laboratory work can be assessed by good PG demonstrators. However, these assessments must be made in a framework laid down by academics. Students can also assess each other under carefully controlled conditions" Some who disagreed said:

"Self-assessment is one good solution, assessment by other students is another"
"If one of the skills we expect in graduates is to be able to communicate effectively, then communication with a known audience (academic staff) but an unknown reader must surely be a reasonable test? I would hate to see a student only assessed by strangers, and only assessed by his/her tutor. What about tutor bias? A former colleague of mine always overmarked public school boys and leggy blonde women!"
"Tutors will always have greater familiarity with students, but part of the exercise is communication and not necessarily to someone already 'on side'"
"The tutor/instruction is the seat of professional information. It is very necessary that each institution has an assessment team or department. The task of this team or department is to develop the appropriate assessment vehicles for the subject/program to be assessed."

Survey Questions

6. It isn't possible to assess students fairly in groups, because the idle get carried by the rest and the best students get their marks pulled down by the weakest

The number of those agreeing with the statement was only five, with fourteen disagreeing.

Of those who agreed, sample statements included:

"One can assess task-sharing dynamics, co-operative behaviour and that is about it."
"Correct, some form of confidential self-assessment and grassing up your mates might compensate"
"We can assess people in groups, certainly, but degrees are awarded to individuals and it is here we must concentrate our assessment"
"This is the perennial comment from students themselves. However, as group work is an essential part of LEARNING, the solution is to divorce assessment from the group work"

Some who disagreed said:

"Big subject and it depends on the nature of the group exercise. Incorporation of peer-assessment, vivas and individually assessed items in an overall mix helps. However, it is true that there is likely to be an averaging-out. If leadership is being assessed (v. hard) or other personal qualities, it gives an opportunity for different aspects of student performance to be assessed and awarded"
"Self-assessment by the group and by individual members of the group is one solution. Also, examiners can observe the group and discuss with the group"
"In my experience, students do not give marks to the passengers!!"
"Knowledge is always social. We must use more time in student groups to discuss the goal of studying. If students can formulate the goals of working by themselves, we do not have to give them any marks. They just do their work, and we can assess the process"

Survey Questions

7. The introduction of modular and semesterised university systems has irreparably damaged the student assessment process

Respondents were more or less evenly ranged on the statement with ten agreeing and nine disagreeing.

Comments in agreement included:

"I think it has tended to trivialise assessments by giving students too much to do in too short a time. If they have too many assessments, they have to choose which ones to spend time on and which to let go, and hope everything is OK. If your module is core, they will treat it seriously; if it's an option or elective, the standard will be less"
"It seems to be necessary from the lecturer's point of view to assess each module. This has led to too many exams and over-assessment"
"Well, in the short term, staff have not shown much creativity in the assessment process. In my institution, many staff have stopped using coursework, and gone back to exams. I think this shows a lack of creative thinking. I personally have taken the opportunity to introduce new ways of looking at assessment"

People who disagreed said:

"These changes have just made teachers look seriously at the purpose, role and process of assessment"
"Badly damaged, yes. Mainly because standardisation of assessment has led to lack of imagination and driven a system always prone to over-assessment into assessment overdrive"
"There is nothing fundamentally wrong with either modules or semesters - it is the way that they are designed that seems to be causing problems in many instances. In my institution, there has been a very worrying decline in the amount of assessment that is formative, and students often feel very poorly prepared for the summative assessments they are required to do"

Survey Questions

8. Technology can help us assess students better

Only five disagreed with the statement with fourteen agreeing.

Among those who agreed, were the following comments:

"I don't know enough about it, but the computer whiz kids here tell me it can be done"
"As much as I detest multiple-choice questions, having students answer such questions (whether in an exam or course evaluation) on a Scantron sheet, enables the instructor to receive the answers back in a fraction of the time it takes"
"The obvious answer is in the marking of mcq papers, but I believe the saving to be relatively marginal, given the extra effort required to build up the question bank. Most other computer-aided assessment schemes I've seen are OK for formative assessment in limited circumstances, but unreliable for summative assessment. They are too open to cheating"

The comments in disagreement included:

"Not better, but differently. It can alleviate the drudgery of record-keeping and mark manipulation. Are we assessing the use of technology?"
"Show me the research. What are we going to assess, some formal, rigid course curriculum which I would argue against?"
"This is very much a matter of discipline. Technology is neutral and it may or may not be apposite to use it according to case"

Survey Questions

9. Other comments and issues for discussion

These included:

"Quality control and standards"
"The connection between assessment and learning, and how much you remember in your 'long-run memory'"
"How do we ensure in a modular programme that each student uses a variety of assessment processes which assess their professional skills, common skills, and use of study skills"
"Standardised assessment versus competency/criterion-referenced assessment"
"Assessing courses in higher education - creative ways of assessing students' learning"
"Avoiding stifling motivation by imposing rigid assessment procedures"
"The role of assessment as a positive integral part of the learning experience. Hence the importance of rapid, detailed, accurate feedback"

Survey Questions

List of respondents

Chris Alder, Electronic and Electrical Engineering, Bradford University, UK
Chris Ashton, Law Dept, Napier University, Edinburgh, UK
John Bradbeer, Dept of Geography, Portsmouth University, UK
Shiona McDonald, St Andrew's College, Glasgow, UK
Ray Thompson, Learning and Teaching Institute, Sheffield Hallam University, Sheffield, UK
Finn Weidersheim-Paul, Dept of Business Studies, University of Uppsala, Sweden
Gill Young, Wolfson school of Health Sciences, Thames Valley University, UK
Enrica Giordano, Instituto di Fisica Applicata, Milan, Italy
Hannu Soini, Dept of Behavioural Studies, University of Oulu, Finland
John Cooper, Dept of Sociology and Psychology, Napier University, Edinburgh, UK
Annie Grant, EDSC, University of Leicester, UK
Bob Matthew, Civil Engineering, University of Bradford, UK
Deborah Street, CSM, University of the West of England, Bristol, UK
John E Thurston, Testing and Assessment Centre, Pima Community College, Tuscon, Arizona, USA
Andrea Toepell, Health studies Programme, Brock University, Ontario, Canada
David Birchall, Henley Management College, UK
Hans-Joachim Pasch, University of Giessen, Germany
Rob Gill, Dept of Behavioural Studies, Huddersfield University, UK
Rick Loy, Dept of Mathematics, Australian National University Canberra, Australia

Survey Questions